Experiences in building and using compiler validation systems

For purposes of this discussion, the term "software validation" refers to the process of testing a completed software product in its operational environment. The scope of this paper is further limited in the following ways: the experiences described are not applicable to applications software; the validation systems discussed must be capable of functioning on a variety of dissimilar hardware and operating systems; the staff performing the validation is not involved in the development or the maintenance of the products being tested; and the result of a validation could impact the eligibility of the product for procurement. This environment imposes unusual and stringent requirements on the portability of the validation systems and the auditability of the validation. The methodologies available for software validation are few. Design simulation 1 can and is being used, concurrently with software system design, to evaluate the effects of various system requirements or design alternatives once the system has been defined and modeled. Problem statement languages l provide a formal syntax and semantics with which to communicate needs of users to analysts, and are applicable during the system definition and design phases of a software development effort. Neither design simulation nor problem statement languages are applicable to the testing of a completed product. The experiences of the Federal COBOL Compiler Testing Service (FCCTS) suggest that functional testing24 is the most thorough technique presently available for testing a completed software product.